SVM Regression Modelling and its Applications

The key method in this thesis is least squares support vector machines (LSSVM), a class of kernel based learning methods that fits within the penalized modelling paradigm. Primary goals of the LS-SVM models are regression and classification. Although local methods (kernel methods) focus directly on estimating the function at a point, they face problems in high dimensions. Therefore, one can guarantee good estimation of a high-dimensional function only if the function is extremely smooth. We have incorporated additional assumptions (the regression function is an additive function of its components) to overcome the curse of dimensionality. We have studied the properties of the LS-SVM regression when relaxing the Gauss-Markov conditions. It was recognized that outliers may have an unusually large influence on the resulting estimate. However, asymptotically the heteroscedasticity does not play any important role. We have developed a robust framework for LS-SVM regression. It allows to obtain a robust estimate based upon the previous LS-SVM regression solution, in a subsequent step. The weights are determined based upon the distribution of the error variables. We have shown, based on the empirical influence curve and the maxbias curve, that the weighted LS-SVM regression is a robust function estimation tool. We have used the same principle to obtain an LS-SVM regression estimate in the heteroscedastic case. However, the weights are then based upon a smooth error variance estimate. Most efficient learning algorithms in neural networks, support vector machines and kernel based learning methods require the tuning of some extra tuning parameters. For practical use, it is often preferable to have a data-driven method to select these parameters. Based on location estimators (e.g., mean, median, M-estimators, L-estimators, R-estimators), we have introduced robust counterparts of model selection criteria (e.g., Cross-Validation, Final Prediction Error criterion). Inference procedures for both linear and nonlinear parametric regression models in fact assume that the output variable follows a normal distribution. With nonparametric regression, the regression equation is determined from the data. In this case, we relax the normality assumption and standard inference procedures are no longer applicable in that case. We have developed a robust approach for obtaining robust prediction intervals by using robust external bootstrapping methods. Finally, we apply LS-SVM regression modelling in the case of density estimation.